1. Field Of The Invention
The present invention relates to a computer program for testing computer hardware and software applications, and specifically relates to an automated form of testing known as regression testing, utilizing comparisons of data representative of data from visual display devices.
2. Related Art
Computer programs used for testing software applications exist which can receive input data, such as key strokes, which are recognized by the software application. Upon receipt of the input data, these computer programs have the ability to store the input data (into files called test scripts), and to replay them such that the software application functions as though a user were actually typing the input data into the software application. In this way, input data can be repeatedly fed into the software application, with a user creating a test script by entering the input data only once.
By using such a computer program, a software application can be executed using a prepared test script, so as to verify that the software application performs as expected. This can be accomplished by somehow comparing previously stored results with results which were subsequently acquired by replaying the test script through the software application. The fact that the software application performs as expected can also be used as an indication that the hardware on which the software application runs is performing as expected. The combination of the hardware on which the software application runs (including a visual display device) and software application under test running thereupon is often referred to as the System Under Test (SUT). The computer program for testing the SUT will be referred to as Regression Testing (RT) software.
RT software has important functions in the software development industry, where during the process of developing a software application, it is often desirable to run each new version of the software application through a long series of test steps. In addition, in the production of computer hardware, it is also often desirable to test each manufactured unit by running a software application on the hardware through a series of steps.
The conventional method for regression testing of hardware and/or software applications was by having a user manually enter input data each time a new piece of hardware and/or new version of a software application was to be tested. The user would then verify that the new version responded to the input data in the same way that a previous version responded to the same test data. This method of regression testing was replete with difficulties, since it was very time intensive, error-prone, and extremely inadequate for production testing or development purposes. In addition, it did not allow for performance measurements and compatibility comparisons of the software application in real-time.
One device for automating the regression testing process was a device jointly developed by Hewlett Packard Corporation and Telamon Corporation of California called Automan. The RT software on Automan provides for capture of test scripts in the form of keystrokes on a keyboard, movements of a mouse, etc. The RT software also provides for replay of these test scripts, and this device replays the test scripts as captured on the software application. A comparison is then made between the data passed from the computer to the terminal, and the originally input data. However, the RT software on Automan does not perform or receive results of comparisons of the visual display device. Thus, the method employed by the Automan device is not a thorough regression testing method, since it is still possible for the data to be typed in correctly, but for the visual display data to be incorrect.
Some of the earlier RT software programs utilized the concept of fixed pacing, which meant that the input data was sent to the software application on the SUT at a fixed interval, rather than determining when the software application was ready to retrieve the next step. Fixed pacing is undesirable, however, for several reasons. First, it may be important in many instances that the input data reach the software application at the appropriate time (that is, it is not enough to merely buffer the input data). Second, since feedback from the SUT is not being gathered during fixed interval testing, performance measurements of components of the SUT cannot be measured. Automan is an example of a fixed pacing device, in that there is a synchronizing mechanism used in conjunction with the terminal.
In addition to the above-mentioned RT software, attempts at applying software patches or modifying the software application itself to create performance measurement data were unsuccessful, since this changed the normal behavior of the software application in unpredictable ways. These intrusive behavior changes often prohibited accurate performance measurements of the software application.
The inventor's realized that what was needed was RT software for preparing and re-playing a test script for software applications running on an SUT, and comparing visual display data with previously stored visual display data in a non-intrusive manner. Problems identified and encountered by the inventors of the present invention with the implementation of such RT software are:
1. A visual display device having more than one valid state (for example, text and graphics) may often be between states at the time that the visual display data is captured. Transferring, storing, and checking for such transitory states results in a large waste of computer time.
2. In general, the visual display data capturing program must be written to coexist with the application being tested and not to interfere with its normal operation.
3. Data transfer time and comparison time destroy the real time operation of the software application being tested.
4. If several different visual display devices are being tested on the system under test at different times, separate visual display data must be maintained. Updating and managing this data is very time consuming and difficult to automate.
5. The test scripts should be portable. (The intent of this objective is to be able to port test scripts to various operating systems).